External user interface for head worn computing

ABSTRACT

Aspects of the present invention relate to user interface control of a head-worn computer.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of U.S. Non-Provisional applicationSer. No. 14/806,385, filed Jul. 22, 2015, the contents of which areincorporated herein by reference in its entirety.

BACKGROUND Field of the Invention

This invention relates to head worn computing. More particularly, thisinvention relates to external user interfaces used in connection with tohead worn computing.

Description of Related Art

Wearable computing systems have been developed and are beginning to becommercialized. Many problems persist in the wearable computing fieldthat need to be resolved to make them meet the demands of the market.

SUMMARY

Aspects of the present invention relate to the systems and methods ofinteracting with a head-worn computer.

These and other systems, methods, objects, features, and advantages ofthe present invention will be apparent to those skilled in the art fromthe following detailed description of the preferred embodiment and thedrawings. All documents mentioned herein are hereby incorporated intheir entirety by reference.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments are described with reference to the following Figures. Thesame numbers may be used throughout to reference like features andcomponents that are shown in the Figures:

FIG. 1 illustrates a head worn computing system in accordance with theprinciples of the present invention.

FIG. 2 illustrates an external user interface in accordance with theprinciples of the present invention.

FIGS. 3a to 3c illustrate distance control systems in accordance withthe principles of the present invention.

FIGS. 4a to 4c illustrate force interpretation systems in accordancewith the principles of the present invention.

FIGS. 5a to 5c illustrate user interface mode selection systems inaccordance with the principles of the present invention.

FIG. 6 illustrates interaction systems in accordance with the principlesof the present invention.

FIG. 7 illustrates external user interfaces in accordance with theprinciples of the present invention.

FIG. 8 illustrates a pattern recognition system and process inaccordance with the principles of the present invention.

FIG. 9 illustrates a projection system in accordance with the principlesof the present invention.

FIG. 10 illustrates an external user interface adapted to be used with asteering wheel, in accordance with the principles of the presentinvention.

FIG. 11 illustrates a dual screen user interface in accordance with theprinciples of the present invention.

FIG. 12 illustrates a wireless finger mountable controller in accordingto the principles of the present invention.

FIG. 13 illustrates a wireless finger mountable controller with a fingercontact sensor in according to the principles of the present invention.

FIG. 14 illustrates a wireless finger mountable controller with a fingercontact sensor in according to the principles of the present invention.

While the invention has been described in connection with certainpreferred embodiments, other embodiments would be understood by one ofordinary skill in the art and are encompassed herein.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT(S)

Aspects of the present invention relate to head-worn computing (“HWC”)systems. HWC involves, in some instances, a system that mimics theappearance of head-worn glasses or sunglasses. The glasses may be afully developed computing platform, such as including computer displayspresented in each of the lenses of the glasses to the eyes of the user.In embodiments, the lenses and displays may be configured to allow aperson wearing the glasses to see the environment through the lenseswhile also seeing, simultaneously, digital imagery, which forms anoverlaid image that is perceived by the person as a digitally augmentedimage of the environment, or augmented reality (“AR”).

HWC involves more than just placing a computing system on a person'shead. The system may need to be designed as a lightweight, compact andfully functional computer display, such as wherein the computer displayincludes a high resolution digital display that provides a high level ofemersion comprised of the displayed digital content and the see-throughview of the environmental surroundings. User interfaces and controlsystems suited to the HWC device may be required that are unlike thoseused for a more conventional computer such as a laptop. For the HWC andassociated systems to be most effective, the glasses may be equippedwith sensors to determine environmental conditions, geographic location,relative positioning to other points of interest, objects identified byimaging and movement by the user or other users in a connected group,and the like. The HWC may then change the mode of operation to match theconditions, location, positioning, movements, and the like, in a methodgenerally referred to as a contextually aware HWC. The glasses also mayneed to be connected, wirelessly or otherwise, to other systems eitherlocally or through a network. Controlling the glasses may be achievedthrough the use of an external device, automatically throughcontextually gathered information, through user gestures captured by theglasses sensors, and the like. Each technique may be further refineddepending on the software application being used in the glasses. Theglasses may further be used to control or coordinate with externaldevices that are associated with the glasses.

Referring to FIG. 1, an overview of the HWC system 100 is presented. Asshown, the HWC system 100 comprises a HWC 102, which in this instance isconfigured as glasses to be worn on the head with sensors such that theHWC 102 is aware of the objects and conditions in the environment 114.In this instance, the HWC 102 also receives and interprets controlinputs such as gestures and movements 116. The HWC 102 may communicatewith external user interfaces 104. The external user interfaces 104 mayprovide a physical user interface to take control instructions from auser of the HWC 102 and the external user interfaces 104 and the HWC 102may communicate bi-directionally to affect the user's command andprovide feedback to the external device 108. The HWC 102 may alsocommunicate bi-directionally with externally controlled or coordinatedlocal devices 108. For example, an external user interface 104 may beused in connection with the HWC 102 to control an externally controlledor coordinated local device 108. The externally controlled orcoordinated local device 108 may provide feedback to the HWC 102 and acustomized GUI may be presented in the HWC 102 based on the type ofdevice or specifically identified device 108. The HWC 102 may alsointeract with remote devices and information sources 112 through anetwork connection 110. Again, the external user interface 104 may beused in connection with the HWC 102 to control or otherwise interactwith any of the remote devices 108 and information sources 112 in asimilar way as when the external user interfaces 104 are used to controlor otherwise interact with the externally controlled or coordinatedlocal devices 108. Similarly, HWC 102 may interpret gestures 116 (e.g.captured from forward, downward, upward, rearward facing sensors such ascamera(s), range finders, IR sensors, etc.) or environmental conditionssensed in the environment 114 to control either local or remote devices108 or 112.

We will now describe each of the main elements depicted on FIG. 1 inmore detail; however, these descriptions are intended to provide generalguidance and should not be construed as limiting. Additional descriptionof each element may also be further described herein.

The HWC 102 is a computing platform intended to be worn on a person'shead. The HWC 102 may take many different forms to fit many differentfunctional requirements. In some situations, the HWC 102 will bedesigned in the form of conventional glasses. The glasses may or may nothave active computer graphics displays. In situations where the HWC 102has integrated computer displays the displays may be configured assee-through displays such that the digital imagery can be overlaid withrespect to the user's view of the environment 114. There are a number ofsee-through optical designs that may be used, including ones that have areflective display (e.g. LCoS, DLP), emissive displays (e.g. OLED, LED),hologram, TIR waveguides, and the like. In addition, the opticalconfiguration may be monocular or binocular. It may also include visioncorrective optical components. In embodiments, the optics may bepackaged as contact lenses. In other embodiments, the HWC 102 may be inthe form of a helmet with a see-through shield, sunglasses, safetyglasses, goggles, a mask, fire helmet with see-through shield, policehelmet with see through shield, military helmet with see-through shield,utility form customized to a certain work task (e.g. inventory control,logistics, repair, maintenance, etc.), and the like.

The HWC 102 may also have a number of integrated computing facilities,such as an integrated processor, integrated power management,communication structures (e.g. cell net, WiFi, Bluetooth, local areaconnections, mesh connections, remote connections (e.g. client server,etc.)), and the like. The HWC 102 may also have a number of positionalawareness sensors, such as GPS, electronic compass, altimeter, tiltsensor, IMU, and the like. It may also have other sensors such as acamera, rangefinder, hyper-spectral camera, Geiger counter, microphone,spectral illumination detector, temperature sensor, chemical sensor,biologic sensor, moisture sensor, ultrasonic sensor, and the like.

The HWC 102 may also have integrated control technologies. Theintegrated control technologies may be contextual based control, passivecontrol, active control, user control, and the like. For example, theHWC 102 may have an integrated sensor (e.g. camera) that captures userhand or body gestures 116 such that the integrated processing system caninterpret the gestures and generate control commands for the HWC 102. Inanother example, the HWC 102 may have sensors that detect movement (e.g.a nod, head shake, and the like) including accelerometers, gyros andother inertial measurements, where the integrated processor mayinterpret the movement and generate a control command in response. TheHWC 102 may also automatically control itself based on measured orperceived environmental conditions. For example, if it is bright in theenvironment the HWC 102 may increase the brightness or contrast of thedisplayed image. In embodiments, the integrated control technologies maybe mounted on the HWC 102 such that a user can interact with itdirectly. For example, the HWC 102 may have a button(s), touchcapacitive interface, and the like.

As described herein, the HWC 102 may be in communication with externaluser interfaces 104. The external user interfaces may come in manydifferent forms. For example, a cell phone screen may be adapted to takeuser input for control of an aspect of the HWC 102. The external userinterface may be a dedicated UI, such as a keyboard, touch surface,button(s), joy stick, and the like. In embodiments, the externalcontroller may be integrated into another device such as a ring, watch,bike, car, and the like. In each case, the external user interface 104may include sensors (e.g. IMU, accelerometers, compass, altimeter, andthe like) to provide additional input for controlling the HWD 104.

As described herein, the HWC 102 may control or coordinate with otherlocal devices 108. The external devices 108 may be an audio device,visual device, vehicle, cell phone, computer, and the like. Forinstance, the local external device 108 may be another HWC 102, whereinformation may then be exchanged between the separate HWCs 108.

Similar to the way the HWC 102 may control or coordinate with localdevices 106, the HWC 102 may control or coordinate with remote devices112, such as the HWC 102 communicating with the remote devices 112through a network 110. Again, the form of the remote device 112 may havemany forms. Included in these forms is another HWC 102. For example,each HWC 102 may communicate its GPS position such that all the HWCs 102know where all of HWC 102 are located.

Referring to FIG. 2, we now turn to describe a particular external userinterface 104, referred to generally as a pen 200. The pen 200 is aspecially designed external user interface 104 and can operate as a userinterface, such as to many different styles of HWC 102. The pen 200generally follows the form of a conventional pen, which is a familiaruser handled device and creates an intuitive physical interface for manyof the operations to be carried out in the HWC system 100. The pen 200may be one of several user interfaces 104 used in connection withcontrolling operations within the HWC system 100. For example, the HWC102 may watch for and interpret hand gestures 116 as control signals,where the pen 200 may also be used as a user interface with the same HWC102. Similarly, a remote keyboard may be used as an external userinterface 104 in concert with the pen 200. The combination of userinterfaces or the use of just one control system generally depends onthe operation(s) being executed in the HWC's system 100.

While the pen 200 may follow the general form of a conventional pen, itcontains numerous technologies that enable it to function as an externaluser interface 104. FIG. 2 illustrates technologies comprised in the pen200. As can be seen, the pen 200 may include a camera 208, which isarranged to view through lens 202. The camera may then be focused, suchas through lens 202, to image a surface upon which a user is writing ormaking other movements to interact with the HWC 102. There aresituations where the pen 200 will also have an ink, graphite, or othersystem such that what is being written can be seen on the writingsurface. There are other situations where the pen 200 does not have sucha physical writing system so there is no deposit on the writing surface,where the pen would only be communicating data or commands to the HWC102. The lens configuration is described in greater detail herein. Thefunction of the camera is to capture information from an unstructuredwriting surface such that pen strokes can be interpreted as intended bythe user. To assist in the predication of the intended stroke path, thepen 200 may include a sensor, such as an IMU 212. Of course, the IMUcould be included in the pen 200 in its separate parts (e.g. gyro,accelerometer, etc.) or an IMU could be included as a single unit. Inthis instance, the IMU 212 is used to measure and predict the motion ofthe pen 200. In turn, the integrated microprocessor 210 would take theIMU information and camera information as inputs and process theinformation to form a prediction of the pen tip movement.

The pen 200 may also include a pressure monitoring system 204, such asto measure the pressure exerted on the lens 202. As will be described ingreater herein, the pressure measurement can be used to predict theuser's intention for changing the weight of a line, type of a line, typeof brush, click, double click, and the like. In embodiments, thepressure sensor may be constructed using any force or pressuremeasurement sensor located behind the lens 202, including for example, aresistive sensor, a current sensor, a capacitive sensor, a voltagesensor such as a piezoelectric sensor, and the like.

The pen 200 may also include a communications module 218, such as forbi-directional communication with the HWC 102. In embodiments, thecommunications module 218 may be a short distance communication module(e.g. Bluetooth). The communications module 218 may be security matchedto the HWC 102. The communications module 218 may be arranged tocommunicate data and commands to and from the microprocessor 210 of thepen 200. The microprocessor 210 may be programmed to interpret datagenerated from the camera 208, IMU 212, and pressure sensor 204, and thelike, and then pass a command onto the HWC 102 through thecommunications module 218, for example. In another embodiment, the datacollected from any of the input sources (e.g. camera 108, IMU 212,pressure sensor 104) by the microprocessor may be communicated by thecommunication module 218 to the HWC 102, and the HWC 102 may performdata processing and prediction of the user's intention when using thepen 200. In yet another embodiment, the data may be further passed onthrough a network 110 to a remote device 112, such as a server, for thedata processing and prediction. The commands may then be communicatedback to the HWC 102 for execution (e.g. display writing in the glassesdisplay, make a selection within the UI of the glasses display, controla remote external device 112, control a local external device 108), andthe like. The pen may also include memory 214 for long or short termuses.

The pen 200 may also include a number of physical user interfaces, suchas quick launch buttons 222, a touch sensor 220, and the like. The quicklaunch buttons 222 may be adapted to provide the user with a fast way ofjumping to a software application in the HWC system 100. For example,the user may be a frequent user of communication software packages (e.g.email, text, Twitter, Instagram, Facebook, Google+, and the like), andthe user may program a quick launch button 222 to command the HWC 102 tolaunch an application. The pen 200 may be provided with several quicklaunch buttons 222, which may be user programmable or factoryprogrammable. The quick launch button 222 may be programmed to performan operation. For example, one of the buttons may be programmed to clearthe digital display of the HWC 102. This would create a fast way for theuser to clear the screens on the HWC 102 for any reason, such as forexample to better view the environment. The quick launch buttonfunctionality will be discussed in further detail below. The touchsensor 220 may be used to take gesture style input from the user. Forexample, the user may be able to take a single finger and run it acrossthe touch sensor 220 to affect a page scroll.

The pen 200 may also include a laser pointer 224. The laser pointer 224may be coordinated with the IMU 212 to coordinate gestures and laserpointing. For example, a user may use the laser 224 in a presentation tohelp with guiding the audience with the interpretation of graphics andthe IMU 212 may, either simultaneously or when the laser 224 is off,interpret the user's gestures as commands or data input.

FIGS. 3A-C illustrate several embodiments of lens and cameraarrangements 300 for the pen 200. One aspect relates to maintaining aconstant distance between the camera and the writing surface to enablethe writing surface to be kept in focus for better tracking of movementsof the pen 200 over the writing surface. Another aspect relates tomaintaining an angled surface following the circumference of the writingtip of the pen 200 such that the pen 200 can be rolled or partiallyrolled in the user's hand to create the feel and freedom of aconventional writing instrument.

FIG. 3A illustrates an embodiment of the writing lens end of the pen200. The configuration includes a ball lens 304, a camera or imagecapture surface 302, and a domed cover lens 308. In this arrangement,the camera views the writing surface through the ball lens 304 and domecover lens 308. The ball lens 304 causes the camera to focus such thatthe camera views the writing surface when the pen 200 is held in thehand in a natural writing position, such as with the pen 200 in contactwith a writing surface. In embodiments, the ball lens 304 should beseparated from the writing surface to obtain the highest resolution ofthe writing surface at the camera 302. In embodiments, the ball lens 304is separated by approximately 1 to 3 mm. In this configuration, thedomed cover lens 308 provides a surface that can keep the ball lens 304separated from the writing surface at a constant distance, such assubstantially independent of the angle used to write on the writingsurface. For instance, in embodiments the field of view of the camera inthis arrangement would be approximately 60 degrees.

The domed cover lens, or other lens 308 used to physically interact withthe writing surface, will be transparent or transmissive within theactive bandwidth of the camera 302. In embodiments, the domed cover lens308 may be spherical or other shape and comprised of glass, plastic,sapphire, diamond, and the like. In other embodiments where lowresolution imaging of the surface is acceptable. The pen 200 can omitthe domed cover lens 308 and the ball lens 304 can be in direct contactwith the surface.

FIG. 3B illustrates another structure where the construction is somewhatsimilar to that described in connection with FIG. 3A; however thisembodiment does not use a dome cover lens 308, but instead uses a spacer310 to maintain a predictable distance between the ball lens 304 and thewriting surface, wherein the spacer may be spherical, cylindrical,tubular or other shape that provides spacing while allowing for an imageto be obtained by the camera 302 through the lens 304. In a preferredembodiment, the spacer 310 is transparent. In addition, while the spacer310 is shown as spherical, other shapes such as an oval, doughnut shape,half sphere, cone, cylinder or other form may be used.

FIG. 3C illustrates yet another embodiment, where the structure includesa post 314, such as running through the center of the lensed end of thepen 200. The post 314 may be an ink deposition system (e.g. inkcartridge), graphite deposition system (e.g. graphite holder), or adummy post whose purpose is mainly only that of alignment. The selectionof the post type is dependent on the pen's use. For instance, in theevent the user wants to use the pen 200 as a conventional ink depositingpen as well as a fully functional external user interface 104, the inksystem post would be the best selection. If there is no need for the‘writing’ to be visible on the writing surface, the selection would bethe dummy post. The embodiment of FIG. 3C includes camera(s) 302 and anassociated lens 312, where the camera 302 and lens 312 are positioned tocapture the writing surface without substantial interference from thepost 314. In embodiments, the pen 200 may include multiple cameras 302and lenses 312 such that more or all of the circumference of the tip 314can be used as an input system. In an embodiment, the pen 200 includes acontoured grip that keeps the pen aligned in the user's hand so that thecamera 302 and lens 312 remains pointed at the surface.

Another aspect of the pen 200 relates to sensing the force applied bythe user to the writing surface with the pen 200. The force measurementmay be used in a number of ways. For example, the force measurement maybe used as a discrete value, or discontinuous event tracking, andcompared against a threshold in a process to determine a user's intent.The user may want the force interpreted as a ‘click’ in the selection ofan object, for instance. The user may intend multiple force exertionsinterpreted as multiple clicks. There may be times when the user holdsthe pen 200 in a certain position or holds a certain portion of the pen200 (e.g. a button or touch pad) while clicking to affect a certainoperation (e.g. a ‘right click’). In embodiments, the force measurementmay be used to track force and force trends. The force trends may betracked and compared to threshold limits, for example. There may be onesuch threshold limit, multiple limits, groups of related limits, and thelike. For example, when the force measurement indicates a fairlyconstant force that generally falls within a range of related thresholdvalues, the microprocessor 210 may interpret the force trend as anindication that the user desires to maintain the current writing style,writing tip type, line weight, brush type, and the like. In the eventthat the force trend appears to have gone outside of a set of thresholdvalues intentionally, the microprocessor may interpret the action as anindication that the user wants to change the current writing style,writing tip type, line weight, brush type, and the like. Once themicroprocessor has made a determination of the user's intent, a changein the current writing style, writing tip type, line weight, brush type,and the like may be executed. In embodiments, the change may be noted tothe user (e.g. in a display of the HWC 102), and the user may bepresented with an opportunity to accept the change.

FIG. 4A illustrates an embodiment of a force sensing surface tip 400 ofa pen 200. The force sensing surface tip 400 comprises a surfaceconnection tip 402 (e.g. a lens as described herein elsewhere) inconnection with a force or pressure monitoring system 204. As a useruses the pen 200 to write on a surface or simulate writing on a surfacethe force monitoring system 204 measures the force or pressure the userapplies to the writing surface and the force monitoring systemcommunicates data to the microprocessor 210 for processing. In thisconfiguration, the microprocessor 210 receives force data from the forcemonitoring system 204 and processes the data to make predictions of theuser's intent in applying the particular force that is currently beingapplied. In embodiments, the processing may be provided at a locationother than on the pen (e.g. at a server in the HWC system 100, on theHWC 102). For clarity, when reference is made herein to processinginformation on the microprocessor 210, the processing of informationcontemplates processing the information at a location other than on thepen. The microprocessor 210 may be programmed with force threshold(s),force signature(s), force signature library and/or other characteristicsintended to guide an inference program in determining the user'sintentions based on the measured force or pressure. The microprocessor210 may be further programmed to make inferences from the forcemeasurements as to whether the user has attempted to initiate a discreteaction (e.g. a user interface selection ‘click’) or is performing aconstant action (e.g. writing within a particular writing style). Theinferencing process is important as it causes the pen 200 to act as anintuitive external user interface 104.

FIG. 4B illustrates a force 408 versus time 410 trend chart with asingle threshold 418. The threshold 418 may be set at a level thatindicates a discrete force exertion indicative of a user's desire tocause an action (e.g. select an object in a GUI). Event 412, forexample, may be interpreted as a click or selection command because theforce quickly increased from below the threshold 418 to above thethreshold 418. The event 414 may be interpreted as a double clickbecause the force quickly increased above the threshold 418, decreasedbelow the threshold 418 and then essentially repeated quickly. The usermay also cause the force to go above the threshold 418 and hold for aperiod indicating that the user is intending to select an object in theGUI (e.g. a GUI presented in the display of the HWC 102) and ‘hold’ fora further operation (e.g. moving the object).

While a threshold value may be used to assist in the interpretation ofthe user's intention, a signature force event trend may also be used.The threshold and signature may be used in combination or either methodmay be used alone. For example, a single-click signature may berepresented by a certain force trend signature or set of signatures. Thesingle-click signature(s) may require that the trend meet a criteria ofa rise time between x any y values, a hold time of between a and bvalues and a fall time of between c and d values, for example.Signatures may be stored for a variety of functions such as click,double click, right click, hold, move, etc. The microprocessor 210 maycompare the real-time force or pressure tracking against the signaturesfrom a signature library to make a decision and issue a command to thesoftware application executing in the GUI.

FIG. 4C illustrates a force 408 versus time 410 trend chart withmultiple thresholds 418. By way of example, the force trend is plottedon the chart with several pen force or pressure events. As noted, thereare both presumably intentional events 420 and presumablynon-intentional events 422. The two thresholds 418 of FIG. 4C createthree zones of force: a lower, middle and higher range. The beginning ofthe trend indicates that the user is placing a lower zone amount offorce. This may mean that the user is writing with a given line weightand does not intend to change the weight, the user is writing. Then thetrend shows a significant increase 420 in force into the middle forcerange. This force change appears, from the trend to have been sudden andthereafter it is sustained. The microprocessor 210 may interpret this asan intentional change and as a result change the operation in accordancewith preset rules (e.g. change line width, increase line weight, etc.).The trend then continues with a second apparently intentional event 420into the higher-force range. During the performance in the higher-forcerange, the force dips below the upper threshold 418. This may indicatean unintentional force change and the microprocessor may detect thechange in range however not affect a change in the operations beingcoordinated by the pen 200. As indicated above, the trend analysis maybe done with thresholds and/or signatures.

Generally, in the present disclosure, instrument stroke parameterchanges may be referred to as a change in line type, line weight, tiptype, brush type, brush width, brush pressure, color, and other forms ofwriting, coloring, painting, and the like.

Another aspect of the pen 200 relates to selecting an operating mode forthe pen 200 dependent on contextual information and/or selectioninterface(s). The pen 200 may have several operating modes. Forinstance, the pen 200 may have a writing mode where the userinterface(s) of the pen 200 (e.g. the writing surface end, quick launchbuttons 222, touch sensor 220, motion based gesture, and the like) isoptimized or selected for tasks associated with writing. As anotherexample, the pen 200 may have a wand mode where the user interface(s) ofthe pen is optimized or selected for tasks associated with software ordevice control (e.g. the HWC 102, external local device, remote device112, and the like). The pen 200, by way of another example, may have apresentation mode where the user interface(s) is optimized or selectedto assist a user with giving a presentation (e.g. pointing with thelaser pointer 224 while using the button(s) 222 and/or gestures tocontrol the presentation or applications relating to the presentation).The pen may, for example, have a mode that is optimized or selected fora particular device that a user is attempting to control. The pen 200may have a number of other modes and an aspect of the present inventionrelates to selecting such modes.

FIG. 5A illustrates an automatic user interface(s) mode selection basedon contextual information. The microprocessor 210 may be programmed withIMU thresholds 514 and 512. The thresholds 514 and 512 may be used asindications of upper and lower bounds of an angle 504 and 502 of the pen200 for certain expected positions during certain predicted modes. Whenthe microprocessor 210 determines that the pen 200 is being held orotherwise positioned within angles 502 corresponding to writingthresholds 514, for example, the microprocessor 210 may then institute awriting mode for the pen's user interfaces. Similarly, if themicroprocessor 210 determines (e.g. through the IMU 212) that the pen isbeing held at an angle 504 that falls between the predetermined wandthresholds 512, the microprocessor may institute a wand mode for thepen's user interface. Both of these examples may be referred to ascontext based user interface mode selection as the mode selection isbased on contextual information (e.g. position) collected automaticallyand then used through an automatic evaluation process to automaticallyselect the pen's user interface(s) mode.

As with other examples presented herein, the microprocessor 210 maymonitor the contextual trend (e.g. the angle of the pen over time) in aneffort to decide whether to stay in a mode or change modes. For example,through signatures, thresholds, trend analysis, and the like, themicroprocessor may determine that a change is an unintentional changeand therefore no user interface mode change is desired.

FIG. 5B illustrates an automatic user interface(s) mode selection basedon contextual information. In this example, the pen 200 is monitoring(e.g. through its microprocessor) whether or not the camera at thewriting surface end 208 is imaging a writing surface in close proximityto the writing surface end of the pen 200. If the pen 200 determinesthat a writing surface is within a predetermined relatively shortdistance, the pen 200 may decide that a writing surface is present 502and the pen may go into a writing mode user inteface(s) mode. In theevent that the pen 200 does not detect a relatively close writingsurface 504, the pen may predict that the pen is not currently beingused to as a writing instrument and the pen may go into a non-writinguser interface(s) mode.

FIG. 5C illustrates a manual user interface(s) mode selection. The userinterface(s) mode may be selected based on a twist of a section 508 ofthe pen 200 housing, clicking an end button 510, pressing a quick launchbutton 222, interacting with touch sensor 220, detecting a predeterminedaction at the pressure monitoring system (e.g. a click), detecting agesture (e.g. detected by the IMU), etc. The manual mode selection mayinvolve selecting an item in a GUI associated with the pen 200 (e.g. animage presented in the display of HWC 102).

In embodiments, a confirmation selection may be presented to the user inthe event a mode is going to change. The presentation may be physical(e.g. a vibration in the pen 200), through a GUI, through a lightindicator, etc.

FIG. 6 illustrates a couple pen use-scenarios 600 and 601. There aremany use scenarios and we have presented a couple in connection withFIG. 6 as a way of illustrating use scenarios to further theunderstanding of the reader. As such, the use-scenarios should beconsidered illustrative and non-limiting.

Use scenario 600 is a writing scenario where the pen 200 is used as awriting instrument. In this example, quick launch button 122A is pressedto launch a note application 610 in the GUI 608 of the HWC 102 display604. Once the quick launch button 122A is pressed, the HWC 102 launchesthe note program 610 and puts the pen into a writing mode. The user usesthe pen 200 to scribe symbols 602 on a writing surface, the pen recordsthe scribing and transmits the scribing to the HWC 102 where symbolsrepresenting the scribing are displayed 612 within the note application610.

Use scenario 601 is a gesture scenario where the pen 200 is used as agesture capture and command device. In this example, the quick launchbutton 122B is activated and the pen 200 activates a wand mode such thatan application launched on the HWC 102 can be controlled. Here, the usersees an application chooser 618 in the display(s) of the HWC 102 wheredifferent software applications can be chosen by the user. The usergestures (e.g. swipes, spins, turns, etc.) with the pen to cause theapplication chooser 618 to move from application to application. Oncethe correct application is identified (e.g. highlighted) in the chooser618, the user may gesture or click or otherwise interact with the pen200 such that the identified application is selected and launched. Oncean application is launched, the wand mode may be used to scroll, rotate,change applications, select items, initiate processes, and the like, forexample.

In an embodiment, the quick launch button 122A may be activated and theHWC 102 may launch an application chooser presenting to the user a setof applications. For example, the quick launch button may launch achooser to show all communication programs (e.g. SMS, Twitter,Instagram, Facebook, email, etc.) available for selection such that theuser can select the program the user wants and then go into a writingmode. By way of further example, the launcher may bring up selectionsfor various other groups that are related or categorized as generallybeing selected at a given time (e.g. Microsoft Office products,communication products, productivity products, note products,organizational products, and the like).

FIG. 7 illustrates yet another embodiment of the present invention. FIG.7 illustrates a watchband clip on controller 700. The watchband clip oncontroller may be a controller used to control the HWC 102 or devices inthe HWC system 100. The watchband clip on controller 700 has a fastener718 (e.g. rotatable clip) that is mechanically adapted to attach to awatchband, as illustrated at 704.

The watchband controller 700 may have quick launch interfaces 708 (e.g.to launch applications and choosers as described herein), a touch pad714 (e.g. to be used as a touch style mouse for GUI control in a HWC 102display) and a display 712. The clip 718 may be adapted to fit a widerange of watchbands so it can be used in connection with a watch that isindependently selected for its function. The clip, in embodiments, isrotatable such that a user can position it in a desirable manner. Inembodiments the clip may be a flexible strap. In embodiments, theflexible strap may be adapted to be stretched to attach to a hand,wrist, finger, device, weapon, and the like.

In embodiments, the watchband controller may be configured as aremovable and replaceable watchband. For example, the controller may beincorporated into a band with a certain width, segment spacing's, etc.such that the watchband, with its incorporated controller, can beattached to a watch body. The attachment, in embodiments, may bemechanically adapted to attach with a pin upon which the watchbandrotates. In embodiments, the watchband controller may be electricallyconnected to the watch and/or watch body such that the watch, watch bodyand/or the watchband controller can communicate data between them.

The watchband controller may have 3-axis motion monitoring (e.g. throughan IMU, accelerometers, magnetometers, gyroscopes, etc.) to capture usermotion. The user motion may then be interpreted for gesture control.

In embodiments, the watchband controller may comprise fitness sensorsand a fitness computer. The sensors may track heart rate, caloriesburned, strides, distance covered, and the like. The data may then becompared against performance goals and/or standards for user feedback.

Another aspect of the present invention relates to tracking penmovements with the assistance of a camera and displayed content in a HWC102. In embodiments, content is presented in a see-through display of ahead-worn computer to provide a virtual guide for the wearer who wantsto make motions with a pen, finger, or other interface and have themotions interpreted for pattern recognition. As described in connectionwith pen embodiments disclosed herein elsewhere, an IMU or pen-tipcamera may be used to monitor the motion of a pen in order to predictwhat patterns are being drawn. The IMU and/or pen tip camera may sufferfrom electronic or optical drift and the drift may cause inaccuracies inthe pattern prediction. In embodiments, to augment the IMU and/or pentip camera motion predictions the virtual guide is provided tocompensate for the drift. The pen motions may be captured by a cameraon-board the HWC 102 while the wearer is writing with the guidance ofthe virtual line. Knowing that the wearer is using the virtual line as aguide, the relative position between the pen tip and virtual line can beused to reduce or eliminate drift issues.

In embodiments, digital content is presented to a wearer of the HWC 102and the wearer moves the pen 200 along a writing surface guided by thedigital content for pattern recordation, recognition and presentationassistance. In embodiments, a camera in the HWC 102 images and tracksthe positions of the pen 200 for pattern recordation and recognitionassistance. In embodiments, both the digital content and the cameracapturing the pen positions are used for pattern recordation andrecognition assistance. In embodiments, the digital content, cameracapture, in-pen camera capture, in-pen IMU, etc. may be used incombination for pattern recordation and recognition assistance. Inembodiments, the relative positions of the pen strokes to the virtualline may be presented in the HWC 102 displays in relation to the virtualline. For example, the wearer of the HWC 102 may be scribing without inkin relation to the virtual line that he perceives and as presented inthe HWC 102 display, the on-board HWC 102 camera may capture thescribing, a processor may interpret the imaged scribing in relation tothe line such that the scribing can be converted into digital content tobe displayed in the HWC 102 display in relation to the virtual line.

FIG. 8 illustrates a system where a camera in the HWC 102 is used totrack pen 200 motions and digital content is presented to the wearer ofthe HWC 102 to assist the wearer with writing within a structure. Inthis embodiment, digital content in the form of a line 804 is presentedin an FOV 802 of the HWC 102. The wearer can see through the FOV 802 sothe line 804 appears to augment the surrounding environment's view forthe wearer. The line may be ‘fixed’ to a spot in the environment suchthat when the wearer turns his head and hence changes the position ofthe HWC 102, the line appears to stay in position with respect to theenvironment. In embodiments, the camera in the HWC 102 may image theenvironment and track the relative movement of the HWC 102 with respectto the environment such that the line 804 can be positioned and movedwithin the FOV in accordance with the imaged movements to maintainvisual alignment of the line with a point, object, marker, etc. in theenvironment. This configuration presents a virtual line in theenvironment that does not appear to move as the wearer's head moves. Thevirtual line can provide the wearer with guidance on where to make penstrokes. The line can be thought of as a line on a piece of paper so thewearer can write, or make strokes in a writing pattern, along thevirtual line to make prediction of the lines pattern more accurate andovercome drift errors that may otherwise be apparent when attempting torecord the movements and predict the patterns.

With the virtual line presented and virtually connected to a position inthe environment, the wearer can use the line for guidance when makingwriting patterns. The HWC 102 camera can also be used to track themovements of the pen 200 relative to the position of the virtual line.This may be used to better predict the patterns indicated by thewearer's pen strokes. As described herein elsewhere, the pen 200 maytrack its motions through a pen tip camera and IMU. In embodiments, thepen tip camera and IMU may track the pen's motion and the camera may beused to track the motion of the pen relative to the virtual line. Eachof these inputs may be used to track, record and predict what it beingwritten.

In embodiments, the camera in the HWC 102 captures images of thewearer's pen's motion while the wearer is using the pen to make patternswith the virtual line as a guide. The virtual line may then be overlaidon the captured images of the motion to assist with the patternanalysis. In embodiments, once the overlay is made, one can see oranalyze how the pen pattern moved with respect to the position of thevirtual line as the wearer may be viewed the virtual line. The patternanalysis may involve interpreting the IMU motion detection, in-penmotion detection, and/or the pen's motion as captured through the HWC102 camera relative to the virtual line. For example, if the IMUindicates that the pen shifted away from the wearer but the position ofthe pen relative to the virtual line indicates the pen was not moving,the portion of IMU data that indicated the shift may be discounted inthe prediction analysis. The virtual line pattern analysis may be donein real-time, after the fact, etc. The pattern recognition may be doneon a processor on-board the HWC 102, remote from the HWC 102, orpartially on-board and remotely.

In embodiments, the virtual line may take any number of forms. Forexample, the virtual line may be a line, part of a virtual note, part ofa virtual message template, etc. The line may also change positions andshapes depending on the wearer's needs. For example, the wearer may wantto trace a pattern that is being displayed as digital content and thedigital content may be presented as a consolidated image, part of animage, image in a line-by-line presentation format, etc. In embodiments,this system may be used for lessons on writing, painting, drawing, etc.

Another aspect of the present invention relates to the projection ofimagery from a head-worn computer, wherein a projector with x-y mirrorcontrol and a solid state lighting system are mounted in the head-worncomputer and positioned to project a raster style image onto a nearbysurface.

FIG. 9 illustrates a projection system according to the principles ofthe present invention. In embodiments, the HWC 102 has a micro-mirrorprojector 902 adapted to project a raster style beam of light togenerate an image on a nearby surface. The micro-mirror projector 902may include two movable mirrors for the x-y directional control of thelight source or a single mirror that has movement on two independentlycontrolled perpendicular axes for x-y directional control. The lightsource may be a monochromatic, multi-chromatic, dual color, tri color,multi-colored, or other arrangement. In the embodiments where multiplecolors are used (e.g. red, green, and blue) the colors may besequentially provided, simultaneously provided, or otherwise provided bya solid-state lighting system (e.g. LED, Laser, etc.). It should beunderstood that the term “raster” is being used herein as an example ofa pattern that may be projected to produce the image on the nearbysurface and it should not be considered limited to any one particularpattern unless otherwise stated. Further, while embodiments refer to a“nearby surface” it should be understood that this is also an examplefor the reader and that it should not be considered limited to anyparticular distance unless otherwise stated.

The micro-mirror projector 902 may project an image for display (e.g. amap, presentation, etc.), an interactive user interface for the HWC 102(e.g. an interactive keyboard, cursor control interface, button, touchpad, etc.), an interactive user interface for an external device 108,interactive content for multiple participants (e.g. a map, game, etc.).

In embodiments, an interactive user interface may be projected by themicro-mirror projector 902 and a sensor system may be included in theHWC 102 to make interpretations related to the intersection of theperson with the image. For example, the sensor system may detect thatthe person has ‘touched’ the letter “a” on a projected keyboard andprovide the detection information to a processor that determines thatthe person has ‘pressed’ the letter “a.”

In embodiments, a sensor system may be included in the HWC 102 adaptedto sense an interference when there is an object between themicro-projector 902 and a nearby surface and the interferenceinformation may be used to modify the projected image such that theprojected image does not project onto the object. For example, themicro-mirror projector 902 may project a keyboard onto a nearby surfaceand when the user places their fingers over the keyboard the sensorsystem can detect the interference from the fingers and the projectioncan be modified such that the projection is dark, not emitted, in thearea of the interference so the user's finger or hand does not have aprojected image on it. In an example, the micro-projector can projectthe keyboard onto the surface. An image of the projected keyboard iscaptured by the camera in the HWC and this image is used as a baselinefor comparison. Images are then captured periodically of the keyboardand compared to the baseline to determine whether fingers are presentand where the fingers are located. The image of the projected keyboardis then modified to remove the portions of the keyboard where thefingers have been determined to be located.

In embodiments, the solid state light used in the micro-mirror projectoris a non-visible laser or LED (e.g. NIR, IR), such that the projectorprojects an invisible image. The invisible image may be detected throughthe use of a matching non-visible light detector or camera. This may beused to prevent others from seeing what the user of the HWC 102 isseeing by providing the HWC 102 with the detector and then displayingvisible image content in the see-through display of the HWC 102 thatmatches the non-visible radiation. This may also be used to project animage for someone else to see if they have a matching non-visibledetector system.

In embodiments, the position of the projected image from themicro-mirror projector 902 is controllable through the HWC 102. Theposition, for example, may be settable through a user gesture, externalcontrol device, HWC 102 mounted interface, etc. In embodiments, theposition is locked in place on the nearby surface. For example, oncepositioned, an adjacent object or a marker on or proximate the nearbysurface may be used to ‘key’ the projected image to such that theprojected image maintains a relative position on the nearby surface. Inthis case, the projected image is stabilized relative to the key, sothat as the HWC moves, the projected image is moved within the displayfield of view to maintain a constant relative position to the key, thisis also known as a world-locked image.

In embodiments, the micro-mirror projector 902 has an imagestabilization system. The image stabilization system may move themicro-mirror projector 902 to compensate for sensed vibrations or othermovements of the HWC 102 such that the projected image appears stablypositioned on the nearby surface even when there are vibrations ormovements (e.g. small movements) of the HWC 102.

An aspect of the present invention relates to a gyro-stabilized imageprojector with gimbaled mounts to provide a physically stabilizedprojection platform. In embodiments, the projection is world locked suchthat the projected image appears in a fixed position relative a surface,surface edge, marker, etc. The world locked projection maybegyro-stabilized with a laser rasterized projection where an IMU is usedto measure movements of the head-worn computer and then small motors(e.g. piezo electric motors) are used to stabilize the projector/rastermirror(s). In an alternate embodiment, the projected image passesthrough optics that include optical stabilization wherein the positionof optical elements can be laterally moved to change the pointingdirection of the projector in response to sensed movements of the HWC.

In addition to optical stabilization, the projected user interface mayalso be, or instead be, digitally stabilized. The digitally generatedimage of the projected user interface can be digitally shifted laterallyacross the projected field of view to stabilize the user interface asseen by the user. Provided the projected user interface (e.g. thekeyboard) only occupies a portion of the projected field of view. Forexample, where the keyboard occupies 20 degrees of a 30 degree projectedfield of view, the projected image can be digitally stabilized formovements of +/−5 degrees by digitally shifting the image to compensatefor detected movements. In embodiments, movement detection can beaccomplished by detecting movements of the HWC or by detecting movementsof objects in the camera's field of view or through a combination ofboth techniques.

Another aspect of the present invention relates to generating theprojected image through a diffractive optical element. In embodiments,the IMU stabilized laser is arranged such that the user interface imageis generated by the diffractive. The laser may be attached to thediffractive and the combined device may be pointed by actuators to alignand stabilize the image. In embodiments, the diffractive is removableand replaceable such that the user can change what projected image is tobe presented by the projector. For example, the HWC 102 may be providedwith a set of diffractives, one for a keyboard, button, slider, etc. andeach one may be removed and replaced in the HWC 102.

An aspect of the present invention relates to a projected or augmentedreality content displayed user interface position and focal plane alongwith the focal plane for content resulting from an interaction with theuser interface. For example, as described herein, a keyboard or otheruser interface may be projected from an HWC 102 onto a surface. The usermay interact with the image that appears on the surface and the HWC 102may have an interaction identification system (e.g. structured invisiblewavelength light pattern recognition system, motion and distance sensorsystem, etc.) such that the interactions produce output (e.g. keystrokes relating to keyboard interactions). The output, or response tothe interactions, may be displayed in the see-through display of the HWC102 at a position and at a focal plane that is in relation to theposition and focal plane of the surface and area where the projecteduser interface is displayed. In embodiments, the position may be suchthat the resultant content does not overlap the user interface from theuser's perspective. In embodiments the focal plane for the presentationof the resultant content and the user interface display surface may bedifferent to form a workspace where the user can either focus on theuser interface or the resultant content, but not both simultaneously. Inembodiments, the position of the resultant content may be world-lockedin relation to the projected user interface such that they appear tomaintain a constant positional relation to one another from the user'sperspective. This can be a useful arrangement for user's that are touchtypers where they focus mainly on the resultant content but occasionallywant to view the keyboard.

In other embodiments, the resultant content may be positioned and lockedin a position near the projected or displayed user interface and havethe same or similar focal plane as the user interface. This arrangementmay be desirable for those users who like to look back and forth betweenthe resultant content and the keys of a keyboard, for example.

In embodiments, the user may affirmatively control the position andfocal plane of the resultant content relative to the projected ordisplayed user interface. The selections may be set as default settings,temporary settings, contextual settings (e.g. a selection based on theapplication being used in the HWC 102, a selection based on the surfacein use for the reference projection or display, time of day, sensorfeedback (e.g. if a motion sensor identifies motion a certain settingmay be used), environmental conditions, etc.

In embodiments, the resultant content may be presented through a displayother than the head-worn see-through display. For example, the user maywant to display the content to other people, either proximate or remotefrom the user, so the user may elect to cause the resultant content tobe presented on another system display.

Another aspect of the present invention relates to maintaining a propershape of a projected or display user interface during movements of theHWC 102. In embodiments, the user interface is presented as aworld-locked item, meaning it is positioned through a fixed reference tosomething in the surrounding environment such that it appears to belocked in place even as the user moves his head and eyes. Inembodiments, the user interface is also stabilized such that relativelysmall movements of the user's head don't cause the user interface toappear to shake or move in an unwanted fashion from the user'sperspective. In a further embodiment, the shape of the projected userinterface may be monitored and adjusted to maintain its intended shapeas to be viewed from the user's perspective (e.g. to correct forkeystoning). For example, the surface or edges of the surface may bemonitored for shape alignment and when the HWC 102 moves enough to causethe shape to otherwise change with respect to the surface reference, theprojected user interface shape may be altered to maintain the properlyaligned shape. In embodiments, active surface alignment may beaccomplished through an imaging process where the camera in the HWC 102is used to image the surface. In embodiments, the shape modificationsmay be accomplished based on a predictive system. For example, an IMUmay monitor the movements of the HWC 102 and the IMU output may be usedto predict the resulting changes in a projected user interface imagesuch that the user interface image can be reshaped based on themovements. In embodiments, the shape management may involve both surfaceimaging and motion-based predictions. In embodiments, the projected ordisplayed user interface may further be digitally stabilized.

Another aspect of the present invention relates to cutting away aportion or all of a displayed or projected user interface based onmovements of the HWC. In embodiments, the user interface is world lockedand either a portion or the whole user interface will be cut off if theHWC 102 moves too much. For example, in the situation where a keyboardis being projected onto a surface and the keyboard is world-locked tothe surface, a portion of the projected keyboard may be eliminated whenthe user turns his head to the side. This prevents the projector fromprojecting the image erroneously. The projector will only have a certainrelatively small adjustable range to target the surface and once the endof the range is reached, for example, the projection can stop or theprojected image can be altered in such a way that only a portion of theprojected user interface still appears. When projecting a portion of theuser interface in such a situation the content being projected may needto be altered. For example, as the right side of the projection isgetting cut off, the digital content may be altered such that the leftportion of the content continues to appear clear.

Another aspect of the present invention relates to world-locking aprojected or displayed user interface based on a user presented marker.In embodiments, a user of a HWC 102 places a high contrast marker ormakes a high contrast mark such that the HWC 102 has a reference for theworld-locking of the user interface. In embodiments, the marker or markmay be intended to be used multiple times, such as a mark on a table topwhere the user periodically sits. In embodiments, the marker may beintended as a one time or limited time mark, such as a mark in the sandor on some surface that the user does not frequently visit. Inembodiments, the marker or mark may be used to world-lock the userinterface where the mark is directly associated with the placement for aportion of the user interface. In other embodiments, the marker or markmay be used as a remote reference for which the user interface will bereferenced but which the user interface will not overlap. This can beuseful in situations where the user wants to move the user interface onthe surface. For example, the user interface may be projected in anoriginal position and the HWC 102 may give the user the opportunity tomove the user interface on the world-locking surface. The user may thenuse a gesture, such as touching the user interface projection anddragging it into the preferred position. Then the HWC may continue touse the mark or marker as a reference or if another marker is recognizedas being available the new marker may be used.

In embodiments, the user generated marker is not visible to the humaneye but can be detected by the HWC 102. For example, quantum dot ink orother infrared active material may be used to make a mark that isinvisible to the human eye but is visible when viewed in infrared andthe HWC 102 may include an infrared camera capable of detecting theinfrared light emitted from the mark. The quantum dot ink or otherinfrared active material may fluoresce infrared light in response tovisible light or near infrared light. Similarly, the mark may be visiblein ultraviolet light but be invisible to the human eye, and the HWC thenincludes an ultraviolet camera.

Another aspect of the present invention relates to capturing userinteractions with the projected or displayed user interface through aprojection of structured light and capturing and interpreting changes inthe structured light caused by user movements. In embodiments, thestructured light is projected through a diffractive to generate a knownpattern of light. The structured light projector may be built into theHWC 102, IMU stabilized and coordinated with the user interfaceprojector to maintain an alignment with the projected user interfaceimage. In embodiments, the structured light will cover the userinterface such that physical interactions with the area involving theuser interface can be identified and interpreted. The structured lightis typically not visible to the user because it is can be a very busypattern that would be distracting to the user. In embodiments, the HWC102 includes a non-visible capture system (e.g. IR camera) to capturethe structured light interference patterns.

In embodiments, a user's finger positions can be calibrated into thestructured light system or stereo camera 3D imaging system by having theuser start with all fingers in contact with the surface. By measuringthe positions of the fingers in contact with the surface by using thestructured light system or stereo camera 3D imaging system, a baselineposition of the fingertips when in contact with the surface can beobtained. When a fingertip later reaches this baseline position, it canbe interpreted as having contacted the surface and a keystroke or otherinput can be determined on the projected or displayed keyboard or userinterface. This is particularly advantageous when the structured lightor stereo camera 3D imaging system views the surface and the user'sfingers from a perspective wherein the movements of the user's fingersare away or toward the HWC, because then the movements of the user'sfingers are limited by the surface. The position and angle of thesurface relative to the user interface can also be determined by usingthe structured light system or stereo camera 3D imaging system. Theposition and angle of the surface can be used to more accuratelydetermine the baseline position of the fingers over the entire area ofthe projected or displayed keyboard or user interface so that keystrokescan be more accurately identified.

In embodiments, a displayed or projected user interface (e.g. akeyboard) and other displayed information are provided to the user incorrespondence with the head pose of the user. Wherein the head pose isdetermined by the measured tilt of the HWC as determined by a tiltsensor associated with the HWC. In this case, the displayed or projecteduser interface is only provided when the user's head pose is at aselected angle or range of angles such as when the user's head is tilteddownward by 30 degrees as is typical when using a laptop computer. Whenthe user, raises their head above this tilt angle, the user interface isnot provided and their finger movements are not tracked. If the userthen tilts their head down again, the user interface is once againprovided and their finger movements are tracked to determine theirinteractions with the user interface. In a similar fashion, the userinterface can be provided when the user's head pose is within a selectedlateral angle or range of angles and if the user moves their headlaterally, the user interface is not provided. This method of providingthe user interface only when the user's head pose is within a selectedposition, encourages the user to keep their head still as is typicalwhen a person is typing or interacting with a graphical user interface.Although, it is within the scope of the invention to provide astabilized displayed or projected user interface within the selectedrange of angles. In this way, a simplified method of world locking theuser interface without having to track objects in the environment isprovided. Instead, the method relies on tracking the head pose of theuser within a selected range of angular movements to determine when adisplayed or projected user interface is provided to the user. Themethod can also be provided as a mode that is selected by the user ofthe HWC. In addition, providing the user interface can be combined withthe providing of other information that is displayed in the HWC when theuser interface is not displayed. For example, when the user has theirhead tilted downward, the user interface can be displayed or projectedand when the user tilts their head upward the other information isprovided while the user interface is not provided.

In embodiments, the user interface is projected using non-visible lightwhere the non-visible light is imaged by the user, or other user, andthe user interface is then presented as an augmented reality overlay inthe head-worn display. For example, the structured light could beprovided at 940 nm (e.g. with an LED or laser diode), which can still becaptured by a standard CMOS or CCD camera with the infrared cut filterremoved. The keyboard could then be projected with 808 nm light (e.g.with an LED or laser diode) and captured with a standard camera. Inembodiments, the image may be captured with the same camera and theimages may be image processed to identify the different wavelengthpatterns. In other embodiments, this may be accomplished with twocameras wherein one camera is used to capture the structured light andthe other camera is used to capture the projected image and each camerais blocked from light associated with the other image (e.g. by includinga notch filter that transmits certain wavelengths of light whilereflecting or absorbing other wavelengths of light).

In embodiments, the structured light pattern and projected userinterface are world locked, stabilized, and image shape corrected in acoordinated fashion to maintain proper alignment between the two andsuch that proper identification of user interactions can be identifiedas properly aligned with the user interface elements. For example, boththe structured light projector and user interface projector may bephysically stabilized (e.g. as described herein), digitally stabilized(e.g. as described herein) and shape corrected to compensate for headmovements (e.g. as described herein).

In embodiments, IMU's are attached to the back of a user's hand, finger,and or knuckles to detect finger movements and surface contact bydetecting sharp stops in movement. This can provide a more detectiblekey contact to go along with detection of finger movements with systems.

Another aspect of the present invention relates to capturing userinteractions with a projected or displayed content user interfacethrough 3D imagery of the user's fingers as captured by two separatedcameras mounted on a head worn computer. For example, a camera may bemounted on ends of the front facing side of the HWC 102 (e.g. near theglasses lenses) and the two cameras may simultaneously capture video ofthe user's fingers while the user is interacting with a projected orcontent displayed user interface (e.g. a projected keyboard. or ARcontent displayed. keyboard). As the cameras capture the images, theimages from the separate cameras can be processed to generate a 3D modelof the user's movements such that interactions with the projected ordisplay user interface (e.g. virtual interface) can be determined. Inembodiments, the dual separated cameras may capture other user body partmovements such that they can be interpreted as 3D gesture commands.

Another aspect of the present invention relates to technologies forlaunching a projected or displayed user interface. In embodiments, theuser interface may be activated (e.g. projected or displayed) based onan affirmative user action, contextual information or other information.For example, if the user launches a software application on thehead-worn computer that interoperates with a particular type of userinterface (e.g. a keyboard, button, mouse, touch pad), the userinterface may be presented to the user automatically. In embodiments,the user interface may be presented when appropriate during theexperience with the software application. For example, if the userlaunches an email application, the user may automatically be presentedwith a ‘reader's’ user interface such as a projected or displayed touchpad. The user may use the touch pad to interact with the email programto assist in reading, scrolling, moving to another email, etc. The usermay also use the touch pad to reply or start a new email, which is anaction that may cause the user interface to alter and include a keyboardto facilitate the input of text. In other embodiments, the user may useanother external user interface to launch the projected or displayeduser interface. For example, the user may have a pen or watch interface(as described herein) and the pen or watch may be adapted to launch theprojected or displayed user interface. The user may then use the pen orwatch for certain interactions and then quick launch an additional userinterface (e.g. a projected or displayed keyboard). The user may alsouse a user interface mounted on the head-worn computer to launch theprojected or displayed interface.

Another aspect of the present invention relates to an invisible userinterface that can be viewed by the user of a head-worn computer. Inembodiments, the user interface is an infrared fluorescing printedkeyboard (e.g. printed with quantum dot ink or infrared active ink)wherein the ink fluoresces in the infrared after being exposed tovisible light or near infrared light. The light from the infraredfluorescing printed keyboard, or other user interface or image, can becaptured by an infrared camera or hyperspectral camera in the HWC, alongwith finger movement. Since the printed keyboard is under the user'sfingers, the fingers don't interfere with the keyboard image. Stereocameras or structured light can be used to determine finger movements ashas been discussed previously herein. Examples of suitable infrared inksinclude: http://www.maxmax.com/aXRayIRInks.asp IR1 ink absorbs below 793nm and emits at 840 nm; or http://www.diversifiednano.com/i-series.aspxx-nano IR-783 absorbs in the visible and emits at 783 nm.

In a further embodiment, user interactions with the projected ordisplayed user interface or printed user interface are captured with atime of flight camera system that is associated with the HWC. Where thetime of flight camera projects a short burst of light (e.g. infraredlight) onto the user hands and the associated area of the userinterface. Light reflected from the user's hands and the associated areaof the user interface or keyboard is then captured for a very shortperiod of time by the time of flight camera. The relative distancebetween the time of flight camera and portions of the user's hands andportions of the associated area of the user interface is then determinedfrom the relative brightness of the different portions of the image ofthe scene as captured by the time of flight camera. As such, the time offlight camera provides a depth map of the user's hands and theassociated area of the user interface. Changes in the depth map are usedto determine the movements of the user's hands in relation to the userinterface.

Another aspect of the present invention relates to a vehicle-specificexternal user interface 104. In embodiments, the vehicle-specificexternal (“VSE”) user interface 104 includes a mechanical mountingsystem adapted to mount the VSE interface 104 on the steering wheel ofthe vehicle. The mounting system may secure the VSE interface in aposition that tends to be near the driver's hands, such as on the wheelportion around the 1:00 to 3:00 position or 9:00 to 11:00 position. TheVSE interface may be secured with a Velcro style system, clip, strap,etc. In embodiments, the VSE interface is adapted to provide the driverwith a system for the interaction with the HWC 102 when driving wherethe interactions are intended to enhance the driver's drivingexperience. For example, a driver may preset applications, screens,content sets, etc. for access while he is driving and the VSE interfacemay provide a physical interface for the launching of an application,toggling, switching, or changing applications or screens or contentsets, etc. The presentation of display content controlled by the VSEinterface may involve navigation, vehicle systems, point-of-interestinformation, advertisements, etc. and the driver may be able to switchbetween the applications very quickly through the interaction of abutton or more than one button. In embodiments, the preset screens,content sets, or applications may be launched through dedicated quicklaunch buttons. For example, the navigation application button may be inthe upper right of the VSE interface.

In embodiments, a pre-programmed button or set of buttons may be set toclear the display of the HWC 102 to be free of content or reduce theamount of content that is otherwise displayed to increase the driver'ssee-through view of the surroundings. The button(s) may be set to switchcontent display modes between two pre-determined content types relatingto the vehicle (e.g. switching between pre-set driving applications).The button(s) may be set to change the amount of content-free area inthe field-of view of the HWC 102. The button(s) may be set to movecontent within the field-of-view. The button(s) may be set to change theHWC 102 display brightness and contrast or control other aspects of theHWC 102, such as to change audio volume, sensor settings, etc. Whilemany embodiments refer to the use of “button(s)” it should be understoodthat this is for simplicity in illustration only and that other forms ofuser controllable interfaces are envisioned by the present invention,such as switches, toggles, touch screens, touch pads, etc.

FIG. 10 illustrates several VSE interfaces according to the principlesof the present invention. The VSE interface 1004 is illustrated as beingmounted on the steering wheel 1002 and illustrated in various controlconfigurations. The VSE interface may have hot or launch buttons on theside 1008, front face 1010 or otherwise such that the driver can touchand interact with them while driving. The VSE interface may also have afixed hot button 1012 to perform a dedicated function such as clearingthe display of the HWC 102 of content or limiting the type or amount ofcontent that is permitted to be displayed in the display. The VSEinterface may also have one or more touch pads or screens. A touch pador screen may, for example, be used as a button style interface as wellas a cursor control style interface. The VSE interface may also bevirtually modified with a virtual active layer 1014. The virtual activelayer 1014 may be presented as digital content in the display of the HWC102 and be locked in position with respect to the physical VSE interfacesuch that the driver perceives the virtual content as augmenting thephysical VSE interface. For example, virtual button labels may beprovided as digital content and overlaid or set next to the VSEinterface such that the driver perceives the labels as being associatedwith the buttons. The virtual content may be used in coordination with anew command set. For example, a new command set relating to navigationmay be set on the HWC 102 and a label or image may be set to appear in aposition locked to the VSE interface. In embodiments, there may not be aphysical button and the interaction that causes a control command may beinitiated when the user virtually interacts with the content by touchinga portion of the VSE controller that intersects, from the driver'sperspective through the display, with the virtual content.

An aspect of the present invention relates to a user interface with aquick launch interface adapted to quickly launch an application, portionof an application, function, display control command, head-worn computerfunction, etc. In embodiments, an external user interface for ahead-worn device is provided (e.g. as described herein elsewhere) andthe external user interface includes a button, switch, touch pad, etc.that when actuated (e.g. the button pressed), an action is initiated onthe head-worn computer (e.g. launching or activating a softwareapplication or clearing the see-through display). In embodiments, theexternal user interface may be in a form of a pen, pen attachment,watch, watch attachment, application specific device (e.g. steeringwheel attachment), programmable device, mouse, wireless finger mountedmouse, phone, music player, etc. (some of which are described hereinelsewhere).

As a further example of an external user interface that includes a quicklaunch activation system, a finger mounted wireless controller (alsogenerally referred to as a wireless finger mouse, wireless air mouse orWAM) may be provided. The WAM may include a gyro and/or inertialmovement detection system (e.g. an IMU) and such system may communicatesignals or commands to the head-worn computer based on its movements.This system may be used to interpret gestures, continuously control themovement of a mouse element on the see-through display, control a viewof content being displayed on the see-through display, etc. The WAM mayalso be mechanically adapted to be mounted on a person's finger (e.g.the index finger) such that its buttons and other physical interfacescan be controlled with the person's thumb. The quick launch physicalinterface (e.g. button) may be positioned on the WAM such that the thumbcan actuate it. Once actuated, the program, action, function, etc.associated with the interface may be initiated.

The quick launch system and associated head-worn computer may beconfigured such that quick launch commands are not acted upon ormodified before being executed based on a situation aware system,head-worn computer setting, external user interface setting, etc. Forexample, the head-worn computer may include sensors that collectinformation that may be interpreted to determine an activity (e.g. aforward speed may be calculated and, in a case where the speed is over10 mph, it may be determined that the person is driving in a car), andthe commands may be ignored or modified based on the activity. In theevent that the situation demands a clear view of the surroundings (e.g.driving a car), a quick launch command that would otherwise causecontent to be presented in the see-through display may be ignored or thecontent displayed may be modified to maintain a high degree of seethrough. In certain situations, this can prevent an obscured view by aninadvertent activation of the quick launch command. In a similarfashion, a quick launch button's commands may be altered or otherwiseinterpreted to cause a predetermined action based on the situation orsetting. For example, irrespective of the command associated with thequick launch interface, activation of the interface may cause theclearing of content from the see-through interface when the situationappears to demand a clear view of the surrounding. As describedelsewhere herein, the quick launch interface may be programmed to causethe see-through display to clear or substantially clear (e.g. onlydisplaying content towards an edge of the display such that it is ‘outof the way’ of the surrounding view).

In embodiments, the quick launch system may be adapted to launch anapplication, function, display control command, etc. when the actuatoris interacted with in a particular way and then send a different commandwhen the interaction is terminated. For example, the system may beadapted to cause content to be displayed in the see-through display onlywhen a button is held. When the button is released, the content may beremoved. This system allows the user to only display content when he hasactivated the interface and he can quickly remove the content, byreleasing, when he is done with the content or wants a clear view of thesurroundings. In embodiments, the system may be programmed in reverse(i.e. content is removed with the button is held). The quick launchsystem may be programmable and/or pre-programmed to set which actuationsystem on the external device is used and what the pattern ofinteraction that causes the action is. In embodiment, an actuator may beprogrammed to cause the launch command after the actuator is held for aperiod of time, actuated multiple times (e.g. double click), or otherinteraction pattern.

In embodiments, the quick launch system may have a ‘hold’ function wherea predetermined interaction causes the launch and then a secondpredetermined action causes a cancellation of the launch or modificationof the launch. For example, a double click of the actuator may cause thedisplay of content in the see-through display and a second double clickor a single click may cause the removal of the content from thesee-through display.

Another aspect of the present invention relates to an auto-adaptinggraphical user interface presented on an external user interface device104 adapted for the control of software applications operating on a HWC102. The auto-adapting graphical user interface receives informationfrom the HWC 102 pertaining to one or more applications operating on theHWC 102 and then the graphical user interface is changed to meet therequirements of the application(s) that are running. The adaptation istailored to assist the user in interacting with the application(s). Forexample, if a text-based application is being presented to the user inthe see-through display of the HWC 102, the auto-adaptable graphicaluser interface may be changed to a touch based keyboard. In the eventthat a game is operating on the HWC 102 the auto-adaptable graphicalinterface may change to a game controller. In embodiments, feedback fromsensor systems in the external user interface device 104 may be useddifferently depending on the application operating on the HWC 102. Forexample, when reading on the HWC 102 tilting the angle of the externaluser interface device 104 may cause the page to scroll, but when the HWCis presenting a game, the tilt feedback may change the way an object inthe game is affected.

FIG. 11 illustrates an external user interface device 104 with anauto-adaptable graphic user interface 1102 in bi-directionalcommunication with a HWC 102. As the illustration indicates, a user ofthe HWC 102 can see through to the environment and see the externaldevice 104 with the auto-adaptable graphical user interface at the sametime the user is viewing digital content present on the see-throughdisplays of the HWC 102. The inventors discovered that this arrangementmakes for an intuitive way to control applications operating on the HWC102 while minimizing the amount of user interface information that getspresented in the see-through displays. For example, when controlling atext-based application, the field of view of the HWC 102 does not haveto include any UI elements or the number of elements may be minimizedbecause the user can still see the control elements on the external userinterface device 104. There are instances when the HWC 102 does includeUI control elements even with the auto-adaptable graphical userinterface 1102. For example, if a cursor or other such element may beneeded and the cursor may be presented in the see-through display of theHWC 102 while the adaptable graphical user interface 1102 convertsautomatically into a touch based or movement based mouse.

In embodiments, the external user interface device 104 may be ahand-held device with a touch screen and a processor, wherein theprocessor is adapted to present graphical information on the touchscreen and interpret user interactions with the touch screen (e.g. atouch screed on a phone, tablet, dedicated device, etc.). The hand-helddevice processor further adapted to receive an indication, from thehead-worn computer, of an operating program of out of a plurality ofcomputer programs the head-worn computer processor is currentlyoperating and presenting to the user. The hand-held device processor maybe further adapted to change the graphical information on the touchscreen based on the indication of operating program such that the touchscreen presents a pre-determined user interface on the touch screen thatmeets the needs of the operating program. In embodiments, the HWC hasits own, separate from the processor in the external user controldevice, processor operating the applications on the HWC. In embodiments,the GUI on the external user control device presents indications (e.g.icons) of available programs to be operated on the HWC where a selectionof the indication initiates the corresponding program on the HWC. Inembodiments, the GUI on the external user control device presentsseveral applications that are already running on the HWC such that oneor more may be selected to be interacted with in the HWC. Inembodiments, the GUI on the external user control device provides acontrol element adapted to switch the external user control device backinto a standalone device that no longer controls the HWC. For example,if the external user control device is a phone or other multi-purposedevice (i.e. controller for the HWC and adapted to perform functionsunrelated to the HWC) the control element may change the device backinto a phone.

Another aspect of the present invention relates to a finger mountedexternal user interface (i.e. a type of external user interface 104)with a sensor positioned and adapted to sense the presence of a user'sfinger. The sensor can provide feedback to an on-board processor of thefinger mounted external user interface and the processor can adjust theinterface's functionality based on the presence or non-presence of thefinger. In embodiments, the sensor facilitated system can act like a‘dead man switch’ where the interface stops controlling a related device(e.g. HWC) when no finger is detected. This can prevent unintentionaloperation. For example, if a user of a HWC 102 has connected an fingerinterface to the HWC 102 such that the finger interface controls aspectsof the HWC 102 and the user dismounts the interface and puts it down(e.g. on a table or seat) it won't inadvertently control the HWC 102because no finger will be detected. In embodiments, such a fingercontrol interface may also have a security system such that onlyauthorized users can properly use the interface to control a relateddevice. For example, the finger controller may have a touch pad and thetouch pad may be adapted to image or otherwise read a fingerprint forauthorization. In embodiments, the user may have a predetermined periodto mount the device after proper authorization. For example, the usermay have his finger print authorized and then have ten seconds to mountthe device such that the finger sensor senses the presence of a finger.

A finger mounted user interface device according to the principles ofthe present invention may have a housing adapted to be mounted on afinger of a user; a finger touch sensor positioned to touch the fingerof the user, when the finger mounted user interface is worn by the user;and the finger touch sensor may be in communication with a processor inthe finger mounted user interface, wherein the processor may regulate afunction of the finger mounted user interface based on the finger touchsensor's indication of a presence or non-presence of the finger of theuser.

The processor may be further adapted to communicate control commands toa head-worn computer. The housing may include a strap to further securethe housing to the finger of the user. The processor may be furtheradapted to control a head-worn computer when the finger touch sensor'sindication is that a finger is present. The processor may be furtheradapted to stop controlling a head-worn computer when the finger touchsensor's indication is that a finger is not present. The finger touchsensor may be a capacitively activated sensor, mechanically activatedsensor, proximity sensor, etc. The function regulated by the processormay control a function based on inertial measurements indicative ofmovements of the device. The function regulated by the processor maycontrol a function based on movements of the device.

FIG. 12 illustrates a wireless finger mounted controller 1202 inaccordance with the principles of the present invention. The controller1202 has a housing 1212 mechanically adapted to sit on top of a finger(e.g. the index finger). The housing includes a strap 1210 that can beslipped over the finger to secure the housing 1212 on the finger. Withthe controller 1202 mounted on the user's finger, the user may use histhumb, or other digit, to interact with the various components on thecontroller (e.g. track pad 1204, actuators 1208 a and 1208 b oractuators on the front of the device). The user may also interact withthe controller 1202 and hence generate control signals for a relateddevice (e.g. HWC 102) by moving the controller in 2D or 3D space. Gyros,inertial measurement units (IMUs), etc. in the controller 1202 maydetect user movements to generate the control signals.

FIG. 13 illustrates a side view of the controller 1202 with internalportions in dashed lines. The top of the housing 1212 is also removed inthis view to expose the circuit board on top and some of the internalcomponents. As illustrated, a contact sensor 1302 may extend through thehousing 1212 into the region where the user's finger fits. The contactsensor 1302 may be a capacitive touch sensor, mechanical touch sensor,etc. The contact sensor may be adapted to sense the presence ornon-presence of the user's finger. A processor in the controller 1202may be connected to the contact sensor 1304 such that the processor canalter functionality of the controller 1202 based on the presence ornon-presence of the user's finger. For example, the controller may beshut off or its controlling functions may be turned off if no finger isdetected.

FIG. 14 illustrates another view of the finger controller 1202 showingthe contact sensor 1302 extending through the housing 1212. Inembodiments, a proximity sensor may be positioned inside of the housingsuch that the sensor does not extend through the housing while stillsensing a finger within its proximity.

Although embodiments of HWC have been described in language specific tofeatures, systems, computer processes and/or methods, the appendedclaims are not necessarily limited to the specific features, systems,computer processes and/or methods described. Rather, the specificfeatures, systems, computer processes and/or and methods are disclosedas non-limited example implementations of HWC. All documents referencedherein are hereby incorporated by reference.

1. (canceled)
 2. A device comprising: one or more processors configuredto communicate with an inertial measurement unit mounted to a user'sfinger; wherein the one or more processors are configured to perform amethod comprising: detecting, via a first output of the inertialmeasurement unit, a movement of the user's finger; detecting, via asecond output of the inertial measurement unit, a stop in the movement;determining whether a sharpness of the stop in the movement exceeds athreshold; in accordance with a determination that the sharpness exceedsthe threshold, presenting an input signal corresponding to a position ofthe user's finger; and in accordance with a determination that thesharpness does not exceed the threshold, forgoing presenting the inputsignal.
 3. The device of claim 2, wherein the stop in the movementcorresponds to a contact of the user's finger with a surface.
 4. Thedevice of claim 3, wherein: the one or more processors are furtherconfigured to communicate with a transmissive display, and concurrentlywith detecting the stop in the movement, the surface is visible to theuser via the transmissive display.
 5. The device of claim 2, wherein theinput signal is associated with one or more of a keyboard input signal,a mouse input signal, a button input signal, and a touch pad inputsignal.
 6. The device of claim 2, wherein the input signal comprises agesture input signal.
 7. The device of claim 2, wherein the one or moreprocessors are further configured to communicate with a display, thedisplay configured to present a user interface element to the user inresponse to the input signal.
 8. The device of claim 2, wherein the oneor more processors are further configured to communicate with atransmissive display, and the position of the user's finger correspondsto a location of a keyboard visible to the user via the transmissivedisplay.
 9. A method comprising: detecting, via a first output of aninertial measurement unit mounted to a user's finger, a movement of theuser's finger; detecting, via a second output of the inertialmeasurement unit, a stop in the movement; determining whether asharpness of the stop in the movement exceeds a threshold; in accordancewith a determination that the sharpness exceeds the threshold,presenting an input signal corresponding to a position of the user'sfinger; and in accordance with a determination that the sharpness doesnot exceed the threshold, forgoing presenting the input signal.
 10. Themethod of claim 9, wherein the stop in the movement corresponds to acontact of the user's finger with a surface.
 11. The method of claim 10,wherein, concurrently with detecting the stop in the movement, thesurface is visible to the user via a transmissive display.
 12. Themethod of claim 9, wherein the input signal is associated with one ormore of a keyboard input signal, a mouse input signal, a button inputsignal, and a touch pad input signal.
 13. The method of claim 9, whereinthe input signal comprises a gesture input signal.
 14. The method ofclaim 9, wherein a display is configured to present a user interfaceelement to the user in response to receiving the input signal.
 15. Themethod of claim 9, wherein the position of the user's finger correspondsto a location of a keyboard visible to the user via a transmissivedisplay.
 16. A non-transitory computer-readable medium storinginstructions which, when executed by one or more processors, cause theone or more processors to perform a method comprising: detecting, via afirst output of an inertial measurement unit mounted to a user's finger,a movement of the user's finger; detecting, via a second output of theinertial measurement unit, a stop in the movement; determining whether asharpness of the stop in the movement exceeds a threshold; in accordancewith a determination that the sharpness exceeds the threshold,presenting an input signal corresponding to a position of the user'sfinger; and in accordance with a determination that the sharpness doesnot exceed the threshold, forgoing presenting the input signal.
 17. Thenon-transitory computer-readable medium of claim 16, wherein the stop inthe movement corresponds to a contact of the user's finger with asurface.
 18. The non-transitory computer-readable medium of claim 17,wherein, concurrently with detecting the stop in the movement, thesurface is visible to the user via a transmissive display.
 19. Thenon-transitory computer-readable medium of claim 16, wherein the inputsignal is associated with one or more of a keyboard input signal, amouse input signal, a button input signal, and a touch pad input signal.20. The non-transitory computer-readable medium of claim 16, wherein theinput signal comprises a gesture input signal.
 21. The non-transitorycomputer-readable medium of claim 16, wherein the position of the user'sfinger corresponds to a location of a keyboard visible to the user viathe transmissive display.